Vitto Avatar Workflow And Vision Copy

Documentation for Developers

Vitto Avatar: Development Workflow & Vision

Part 1: The Development Workflow

What We Built

The Vitto Canvas 2D avatar engine follows a pattern we call "Framework-Agnostic Engine + Thin Shell." The idea is that all the complex visual logic (drawing, animation, transitions, particles) lives in pure TypeScript with zero framework dependencies, while the framework layer (Angular) is just a thin wrapper that passes signals in and renders a canvas out.

┌─────────────────────────────────────────────────┐
│              Pure TypeScript Engine              │
│                                                  │
│  moods.ts    renderer.ts    drawing/*            │
│  types.ts    math.ts        behaviors/*          │
│  noise.ts                                        │
│                                                  │
│  Zero Angular imports. Zero DOM assumptions.     │
│  Just Canvas 2D API + requestAnimationFrame.     │
└──────────────────────┬──────────────────────────┘
                       │
              esbuild bundles
                       │
          ┌────────────┴────────────┐
          │                         │
   ┌──────▼───────┐        ┌───────▼────────┐
   │  IIFE Bundle │        │   ESM Bundle   │
   │  (21KB)      │        │   (21KB)       │
   │              │        │                │
   │  For:        │        │  For:          │
   │  - HTML dev  │        │  - Angular     │
   │    harness   │        │    component   │
   │  - Chrome    │        │  - Future      │
   │    injection │        │    React/Vue   │
   │  - Embeds    │        │    wrappers    │
   └──────────────┘        └────────────────┘

The Edit → Build → See Cycle

Every change follows the same loop regardless of who is working or how they're testing:

  ┌──────────────────────────────────────────────┐
  │                                              │
  │  1. EDIT engine TypeScript source files      │
  │     libs/libs-anim/vitto-avatar/src/lib/     │
  │     engine/{moods,drawing,behaviors}/*.ts     │
  │                                              │
  │  2. REBUILD the bundle                       │
  │     npx esbuild ... → vitto-engine.iife.js   │
  │     (< 1 second)                             │
  │                                              │
  │  3. SEE the result                           │
  │     Paco: refresh dev harness or Angular app │
  │     Claude: re-inject into Chrome + screenshot│
  │                                              │
  │  4. ITERATE — go back to step 1              │
  │                                              │
  └──────────────────────────────────────────────┘

The HTML dev harness (vitto-dev-harness.html) contains zero drawing code — it's purely a control panel: mood buttons, property sliders, a state inspector, and a canvas element. It loads the engine from the built bundle via a <script> tag. This means:

Who Changes What

I want to... Edit this file Rebuild needed?
Change a mood's properties (eyes wider, more blush) engine/moods.ts Yes
Change how eyes are drawn (shape, shading) engine/drawing/face.ts Yes
Change particle effects (sparkles, zzz) engine/drawing/effects.ts Yes
Change body/tail animation engine/drawing/body.ts Yes
Change transition speed or easing engine/renderer.ts Yes
Add a new animatable property engine/types.ts + engine/moods.ts Yes
Tweak idle behavior (ear twitch frequency, head bob) engine/behaviors/idle.ts + engine/noise.ts Yes
Add a new dev control (slider, toggle) vitto-dev-harness.html No
Wire avatar to Angular app components/vitto-avatar.component.ts No (Angular compiles from source)

Why This Matters for Your Team

This pattern is portable to any visual component in your monorepo. Charts, loading animations, data visualizations — anything that renders to Canvas can follow the same structure: pure TS engine → esbuild bundle → thin framework wrapper + standalone dev harness. It means:

Part 2: The Groundbreaking Idea

The Gap in Your Current Architecture

After examining your codebase, I found something interesting about how Vitto's emotions currently flow:

Backend (vitto-api)
  │
  ├── EmotionalContextService.getEmotionalArcAdjustment()
  │     → Fetches conversationGenome from backend-ai-api
  │     → Returns emotionalArc: VittoMood[] + toneAdjustment string
  │     → Used ONLY to tune the LLM system prompt
  │
  ├── PersonaService.visualIdentity.moodDefault
  │     → Static mood per persona (e.g., "Friendly" → cheerful)
  │
  └── SSE Stream (message/:taskId/stream)
        → Events: content_delta, phase_start, tool_start, etc.
        → ❌ NO mood events transmitted to frontend

Frontend (vitto-chat)
  │
  ├── VittoChatStreamingService
  │     → Handles SSE events
  │     → ❌ No mood awareness
  │
  ├── MessageAvatarComponent
  │     → Has mood input (defaults to 'neutral')
  │     → ❌ Parent never passes mood — always neutral
  │
  └── VittoMoodService (in vitto-rive-avatar)
        → Full mood state management: debouncing, history, gaze
        → ❌ Nobody calls setMood() with real data

The emotional intelligence exists in your backend but never reaches the user's eyes. The backend analyzes conversation tone, builds emotional arcs, and adjusts persona behavior — but the avatar on screen stays frozen on "neutral" because no mood events travel through the SSE stream to the frontend.

The Idea: Continuous Emotional Expression

Every AI avatar today — including major players — works the same way: pick one of N preset emotions, play that animation, repeat. It's like acting with flash cards. Vitto can do something no one else does.

Your Canvas 2D engine has overrideProperty() — it can set any of 30+ facial properties to any float value independently. It doesn't need to pick a preset. Combined with the emotional intelligence your backend already computes, this opens the door to continuous emotional blending.

How It Works

Step 1: Backend emits mood events in the SSE stream

The backend already runs EmotionalContextService and knows the emotional arc. Instead of only using it for prompt tuning, it also emits a new SSE event type:

// New SSE event type
interface MoodEvent {
  type: 'mood_update';
  mood: string;           // Base mood key: 'cheerful', 'empathetic', etc.
  intensity: number;      // 0.0 - 1.0, how strongly this mood is expressed
  modifiers?: {           // Optional micro-adjustments
    browAngle?: number;   // e.g., +0.15 concern overlay on cheerful
    sweat?: number;       // e.g., 0.2 mild anxiety
    eyeH?: number;        // e.g., 1.2 extra-wide surprise
  };
  trigger?: string;       // Why: 'user_frustration', 'task_complete', 'humor_detected'
}

This is lightweight — a few bytes per event — and piggybacks on your existing SSE infrastructure.

Step 2: Frontend streams mood into the avatar

VittoChatStreamingService picks up mood_update events and feeds them to VittoMoodService, which already has debouncing, minimum hold times, and smooth transitions built in.

Step 3: The engine blends, not switches

Instead of hard-switching to a preset, the renderer:

  1. Transitions to the base mood (e.g., cheerful)
  2. Applies modifiers as property overrides on top
  3. The result is an expression that doesn't exist in any preset

What This Looks Like in Practice

A user says: "I've been trying to fix this bug for three hours and I'm losing it"

Today's Vitto: stays neutral (no mood data reaches the avatar).

Tomorrow's Vitto with discrete moods: switches to "empathetic" (preset #4).

Vitto with continuous expression:

Base: empathetic (soft eyes, gentle tilt)
  + browAngle: -0.2 (slight concern furrow)
  + sweat: 0.15 (subtle anxiety acknowledgment)
  + headTilt: 0.06 (attentive lean)
  + eyeH: 0.85 (slightly wider — paying close attention)

The user sees an avatar that looks genuinely concerned and attentive — not a canned "empathy" animation, but a nuanced blend that mirrors how a real face would respond. The difference is subtle but immediately felt.

Another example — the user just shipped a feature after a long session:

Base: celebratory (star eyes, big smile)
  + intensity: 0.7 (not over the top — context-aware)
  + proud: 0.3 (blend in some pride)
  + bodyBounce: 0.15 (contained excitement)

Vitto doesn't just flash a party animation — it looks genuinely pleased for you.

Why This Is Groundbreaking

Nobody else can do this. Here's why:

  1. Rive/Lottie/Spine avatars can't blend arbitrarily. They're locked to keyframed states. You can crossfade between two states, but you can't say "take the eyes from empathetic and the mouth from cheerful and add 20% concern." Canvas 2D with parametric properties can.

  2. The emotional intelligence already exists in your backend. The conversationGenome with emotionalArc is computed but wasted. Piping it to the frontend through SSE is trivial — the infrastructure is already there.

  3. The engine is already built for it. overrideProperty(), MOOD_NUMERIC_KEYS, parametric interpolation — these aren't hacks we'd need to add. They're core to the current architecture. The engine was designed for arbitrary property values from day one.

  4. It compounds with your AI. As your emotional detection gets smarter (sentiment analysis, tone classification, user history), the avatar automatically gets more expressive. The rendering layer doesn't need to change — just the quality of the mood vectors it receives.

The Competitive Moat

This creates a feedback loop that's hard to replicate:

Better emotional detection
        ↓
More nuanced mood vectors
        ↓
More expressive avatar
        ↓
Users feel more connected to Vitto
        ↓
More engagement data
        ↓
Better emotional detection ← (loop)

Every interaction trains a more emotionally intelligent avatar. Competitors would need to build all three layers (emotional AI + parametric engine + streaming pipeline) simultaneously. You already have two of the three, and the third (SSE mood events) is a small engineering task.

Implementation Roadmap

Phase What Effort Impact
Phase 1 Add mood_update SSE event type to vitto.controller.ts Small Enables everything
Phase 2 Wire VittoChatStreamingService → VittoMoodService Small Avatar starts reacting to conversation
Phase 3 Backend sends base mood per message (from persona + emotional arc) Medium 14 moods come alive
Phase 4 Backend sends modifiers (property-level adjustments) Medium Continuous expression
Phase 5 Fine-tune emotional detection with user feedback signals Large The moat deepens

Phase 1-3 could ship in a sprint. Phase 4 is where it gets truly differentiated. Phase 5 is the long game.

Part 3: What We Built So Far (For New Session Context)

Phase 1 Completion Status: COMPLETE ✓

Phase 1 (idle behaviors + enhanced rendering) is fully implemented and tested. The avatar now displays continuous organic micro-animations, sophisticated eye rendering, and ambient mood feedback at 21KB minified bundle size.

Engine Source Files

libs/libs-anim/vitto-avatar/src/lib/engine/
├── index.ts          — Bundle entry point (exports everything)
├── renderer.ts       — VittoRenderer class: RAF loop, transitions, blink
├── moods.ts          — 14 mood definitions (30+ properties each)
├── types.ts          — MoodProperties, AnimState, DrawContext, RendererConfig
├── math.ts           — lerp, clamp, easeInOutCubic, easeOutBack
├── noise.ts          — Simplex-inspired noise (noise1D, fbm) for organic idle motion
├── drawing/
│   ├── background.ts — Gradient background + mood-tinted ambient glow (pulsing aura)
│   ├── body.ts       — Tail + body with breathing animation
│   ├── head.ts       — Head orchestrator (calls face + effects)
│   ├── face.ts       — Eyes (3-tier dynamic highlights, mood-tinted iris), nose, mouth, whiskers, blush, brows
│   ├── effects.ts    — Particles: sparkles, hearts, ?, !, zzz, sweat
│   └── shapes.ts     — Helper: drawStar
└── behaviors/
    ├── idle.ts       — 8-channel idle behavior system (earTwitchL/R, headTiltIdle, headBobY, whiskerFlutter, tailVariation, bodyShiftX, eyeWideVariation)
    ├── breathing.ts  — Sinusoidal breath offset
    └── saccades.ts   — Micro pupil movements

Angular Integration Files

libs/libs-anim/vitto-avatar/src/lib/
├── components/vitto-avatar/
│   ├── vitto-avatar.component.ts    — Standalone, OnPush, inject()
│   ├── vitto-avatar.component.html
│   └── vitto-avatar.component.scss
├── services/
│   └── vitto-mood.service.ts        — Signal-based, debounced, API-compatible with rive version
└── types/
    └── vitto-avatar.types.ts        — VittoMood enum, config, registry

Dev Tooling

"vitto animation"/
├── vitto-dev-harness.html           — Standalone dev UI with 27 interactive sliders + Copy JSON + Reset to Preset
├── claude-chrome-testing-workflow.md — How Claude tests visually via Chrome injection
├── PROMPT-chrome-visual-testing.md  — Copy-paste prompt for future agent sessions
└── vitto-avatar-workflow-and-vision.md — This document

Current State (Phase 1)

Next Steps: Phase 2 & Phase 3

Phase 2: Expanded Emotional Properties & Dynamic Shadows

Implement additional emotional expression channels:

Phase 3: Wire Mood Events Through SSE Stream

Connect the backend emotional intelligence pipeline to frontend avatar rendering:

The competitive advantage emerges when all three layers (emotional AI detection + parametric rendering + SSE streaming) work together as a continuous feedback loop.